Data processing apparatuses, methods, computer programs and computer-readable media

ABSTRACT

A set of residual elements useable to reconstruct a rendition of a first time sample of a signal is obtained. A set of spatio-temporal correlation elements associated with the first time sample is generated. The set of spatio-temporal correlation elements is indicative of an extent of spatial correlation between a plurality of residual elements and an extent of temporal correlation between first reference data based on the rendition and second reference data based on a rendition of a second time sample of the signal. The set of spatio-temporal correlation elements is used to generate output data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/GB2017/052631, filed Sep. 8, 2017, which claims priority to UKApplication No. GB1615265.4, filed Sep. 8, 2016, under 35 U.S.C. §119(a). Each of the above-referenced patent applications is incorporatedby reference in its entirety.

BACKGROUND OF THE INVENTION Technical Field

This invention relates to data processing apparatuses, methods, computerprograms and computer-readable media.

Background

Compression and decompression of signals is a consideration in manyknown systems. Many types of signal, for example video, audio orvolumetric signals, may be compressed and encoded for transmission, forexample over a data communications network. Signals may also be storedin a compressed form, for example on a storage medium such as a DigitalVersatile Disc (DVD). When such a signal is decoded, it may be desiredto increase a level of quality of the signal and/or recover as much ofthe information contained in the original signal as possible.

Some known systems exploit scalable encoding techniques. Scalableencoding involves encoding a signal along with information to allow thereconstruction of the signal at one or more different levels of quality,for example depending on the capabilities of the decoder and theavailable bandwidth. However, relatively large amounts of informationmay still be stored and/or transmitted, particularly as the usage ofhigher quality, higher definition video becomes more widespread.

SUMMARY

According to a first aspect of the present invention, there is providedan apparatus configured to: obtain a set of residual elements, the setof residual elements being useable to reconstruct a first rendition of afirst time sample of a signal at a relatively high level of quality in atiered hierarchy having multiple levels of quality using a secondrendition of the first time sample of the signal at the relatively highlevel of quality, the second rendition being based on a rendition of thefirst time sample of the signal at a relatively low level of quality inthe tiered hierarchy; generate a set of spatio-temporal correlationelements associated with the first time sample of the signal, the set ofspatio-temporal correlation elements being indicative of an extent ofspatial correlation between a plurality of residual elements in the setof residual elements and an extent of temporal correlation between firstreference data based on the first rendition and second reference databased on a rendition of a second time sample of the signal; use the setof spatio-temporal correlation elements to generate first output data;and use the rendition at the relatively low level of quality to generatesecond output data.

According to a second aspect of the present invention, there is providedan apparatus configured to: receive input data comprising first inputdata based on a set of spatio-temporal correlation elements and secondinput data based on a rendition of a first time sample of a signal at arelatively low level of quality in a tiered hierarchy having multiplelevels of quality; obtain a set of residual elements using the set ofspatio-temporal correlation elements, the set of residual elements beinguseable to reconstruct a first rendition of the first time sample of thesignal at a relatively high level of quality in the tiered hierarchyusing a second rendition of the first time sample of the signal at therelatively high level of quality, the second rendition being based onthe rendition at the relatively low level of quality; and reconstructthe first rendition at the relatively high level of quality using thesecond rendition and the set of residual elements, wherein the set ofspatio-temporal correlation elements are indicative of an extent ofspatial correlation between a plurality of residual elements in the setof residual elements and an extent of temporal correlation between firstreference data based on the first rendition and second reference databased on a rendition of a second time sample of the signal.

According to a third aspect of the present invention, there is provideda method comprising: obtaining a set of residual elements, the set ofresidual elements being useable to reconstruct a first rendition of afirst time sample of a signal at a relatively high level of quality in atiered hierarchy having multiple levels of quality using a secondrendition of the first time sample of the signal at the relatively highlevel of quality, the second rendition being based on a rendition of thefirst time sample of the signal at a relatively low level of quality inthe tiered hierarchy; generating a set of spatio-temporal correlationelements associated with the first time sample of the signal, the set ofspatio-temporal correlation elements being indicative of an extent ofspatial correlation between a plurality of residual elements in the setof residual elements and an extent of temporal correlation between firstreference data based on the first rendition and second reference databased on a rendition of a second time sample of the signal; using theset of spatio-temporal correlation elements to generate first outputdata; and using the rendition at the relatively low level of quality togenerate second output data.

According to a fourth aspect of the present invention, there is provideda computer program comprising instructions which, when executed, causean apparatus to perform a method comprising: obtaining a set of residualelements, the set of residual elements being useable to reconstruct afirst rendition of a first time sample of a signal at a relatively highlevel of quality in a tiered hierarchy having multiple levels of qualityusing a second rendition of the first time sample of the signal at therelatively high level of quality, the second rendition being based on arendition of the first time sample of the signal at a relatively lowlevel of quality in the tiered hierarchy; generating a set ofspatio-temporal correlation elements associated with the first timesample of the signal, the set of spatio-temporal correlation elementsbeing indicative of an extent of spatial correlation between a pluralityof residual elements in the set of residual elements and an extent oftemporal correlation between first reference data based on the firstrendition and second reference data based on a rendition of a secondtime sample of the signal; using the set of spatio-temporal correlationelements to generate first output data; and using the rendition at therelatively low level of quality to generate second output data.

According to a fifth aspect of the present invention, there is provideda computer-readable medium comprising a computer program comprisinginstructions which, when executed, cause an apparatus to perform amethod comprising: obtaining a set of residual elements, the set ofresidual elements being useable to reconstruct a first rendition of afirst time sample of a signal at a relatively high level of quality in atiered hierarchy having multiple levels of quality using a secondrendition of the first time sample of the signal at the relatively highlevel of quality, the second rendition being based on a rendition of thefirst time sample of the signal at a relatively low level of quality inthe tiered hierarchy; generating a set of spatio-temporal correlationelements associated with the first time sample of the signal, the set ofspatio-temporal correlation elements being indicative of an extent ofspatial correlation between a plurality of residual elements in the setof residual elements and an extent of temporal correlation between firstreference data based on the first rendition and second reference databased on a rendition of a second time sample of the signal; using theset of spatio-temporal correlation elements to generate first outputdata; and using the rendition at the relatively low level of quality togenerate second output data.

According to a sixth aspect of the present invention, there is provideda method comprising: receiving input data comprising first input databased on a set of spatio-temporal correlation elements and second inputdata based on a rendition of a first time sample of a signal at arelatively low level of quality in a tiered hierarchy having multiplelevels of quality; obtaining a set of residual elements using the set ofspatio-temporal correlation elements, the set of residual elements beinguseable to reconstruct a first rendition of the first time sample of thesignal at a relatively high level of quality in the tiered hierarchyusing a second rendition of the first time sample of the signal at therelatively high level of quality, the second rendition being based onthe rendition at the relatively low level of quality; and reconstructingthe first rendition at the relatively high level of quality using thesecond rendition and the set of residual elements, wherein the set ofspatio-temporal correlation elements are indicative of an extent ofspatial correlation between a plurality of residual elements in the setof residual elements and an extent of temporal correlation between firstreference data based on the first rendition and second reference databased on a rendition of a second time sample of the signal.

According to a seventh aspect of the present invention, there isprovided a computer program comprising instructions which, whenexecuted, cause an apparatus to perform a method comprising: receivinginput data comprising first input data based on a set of spatio-temporalcorrelation elements and second input data based on a rendition of afirst time sample of a signal at a relatively low level of quality in atiered hierarchy having multiple levels of quality; obtaining a set ofresidual elements using the set of spatio-temporal correlation elements,the set of residual elements being useable to reconstruct a firstrendition of the first time sample of the signal at a relatively highlevel of quality in the tiered hierarchy using a second rendition of thefirst time sample of the signal at the relatively high level of quality,the second rendition being based on the rendition at the relatively lowlevel of quality; and reconstructing the first rendition at therelatively high level of quality using the second rendition and the setof residual elements, wherein the set of spatio-temporal correlationelements are indicative of an extent of spatial correlation between aplurality of residual elements in the set of residual elements and anextent of temporal correlation between first reference data based on thefirst rendition and second reference data based on a rendition of asecond time sample of the signal.

According to an eighth aspect of the present invention there is provideda computer-readable medium comprising a computer program comprisinginstructions which, when executed, cause an apparatus to perform amethod comprising: receiving input data comprising first input databased on a set of spatio-temporal correlation elements and second inputdata based on a rendition of a first time sample of a signal at arelatively low level of quality in a tiered hierarchy having multiplelevels of quality; obtaining a set of residual elements using the set ofspatio-temporal correlation elements, the set of residual elements beinguseable to reconstruct a first rendition of the first time sample of thesignal at a relatively high level of quality in the tiered hierarchyusing a second rendition of the first time sample of the signal at therelatively high level of quality, the second rendition being based onthe rendition at the relatively low level of quality; and reconstructingthe first rendition at the relatively high level of quality using thesecond rendition and the set of residual elements, wherein the set ofspatio-temporal correlation elements are indicative of an extent ofspatial correlation between a plurality of residual elements in the setof residual elements and an extent of temporal correlation between firstreference data based on the first rendition and second reference databased on a rendition of a second time sample of the signal.

According to a ninth aspect of the present invention, there is providedan apparatus configured to: receive input data comprising first inputdata based on a set of correlation elements and second input data basedon a rendition of a first time sample of a signal at a relatively lowlevel of quality in a tiered hierarchy having multiple levels ofquality; obtain a set of residual elements using the set of correlationelements, the set of residual elements being useable to reconstruct afirst rendition of the first time sample of the signal at a relativelyhigh level of quality in the tiered hierarchy using a second renditionof the first time sample of the signal at the relatively high level ofquality, the second rendition being based on the rendition at therelatively low level of quality; and reconstruct the first rendition atthe relatively high level of quality using the second rendition and theset of residual elements, wherein the set of correlation elements isindicative of at least an extent of spatial correlation between aplurality of residual elements in the set of residual elements, andwherein the input data includes data identifying whether the set ofcorrelation elements is indicative of the extent of spatial correlationor whether the set of correlation elements is further indicative of anextent of temporal correlation between first reference data based on thefirst rendition and second reference data based on a rendition of asecond time sample of the signal.

According to a tenth aspect of the present invention, there is provideda method comprising: receiving input data comprising first input databased on a set of correlation elements and second input data based on arendition of a first time sample of a signal at a relatively low levelof quality in a tiered hierarchy having multiple levels of quality;obtaining a set of residual elements using the set of correlationelements, the set of residual elements being useable to reconstruct afirst rendition of the first time sample of the signal at a relativelyhigh level of quality in the tiered hierarchy using a second renditionof the first time sample of the signal at the relatively high level ofquality, the second rendition being based on the rendition at therelatively low level of quality; and reconstructing the first renditionat the relatively high level of quality using the second rendition andthe set of residual elements, wherein the set of correlation elements isindicative of at least an extent of spatial correlation between aplurality of residual elements in the set of residual elements, andwherein the input data includes data identifying whether the set ofcorrelation elements is indicative of the extent of spatial correlationor whether the set of correlation elements is further indicative of anextent of temporal correlation between first reference data based on thefirst rendition and second reference data based on a rendition of asecond time sample of the signal.

According to an eleventh aspect of the present invention, there isprovided a computer program comprising instructions which, whenexecuted, cause an apparatus to perform a method comprising: receivinginput data comprising first input data based on a set of correlationelements and second input data based on a rendition of a first timesample of a signal at a relatively low level of quality in a tieredhierarchy having multiple levels of quality; obtaining a set of residualelements using the set of correlation elements, the set of residualelements being useable to reconstruct a first rendition of the firsttime sample of the signal at a relatively high level of quality in thetiered hierarchy using a second rendition of the first time sample ofthe signal at the relatively high level of quality, the second renditionbeing based on the rendition at the relatively low level of quality; andreconstructing the first rendition at the relatively high level ofquality using the second rendition and the set of residual elements,wherein the set of correlation elements is indicative of at least anextent of spatial correlation between a plurality of residual elementsin the set of residual elements, and wherein the input data includesdata identifying whether the set of correlation elements is indicativeof the extent of spatial correlation or whether the set of correlationelements is further indicative of an extent of temporal correlationbetween first reference data based on the first rendition and secondreference data based on a rendition of a second time sample of thesignal.

According to a twelfth aspect of the present invention, there isprovided a computer-readable medium comprising a computer programcomprising instructions which, when executed, cause an apparatus toperform a method comprising: receiving input data comprising first inputdata based on a set of correlation elements and second input data basedon a rendition of a first time sample of a signal at a relatively lowlevel of quality in a tiered hierarchy having multiple levels ofquality; obtaining a set of residual elements using the set ofcorrelation elements, the set of residual elements being useable toreconstruct a first rendition of the first time sample of the signal ata relatively high level of quality in the tiered hierarchy using asecond rendition of the first time sample of the signal at therelatively high level of quality, the second rendition being based onthe rendition at the relatively low level of quality; and reconstructingthe first rendition at the relatively high level of quality using thesecond rendition and the set of residual elements, wherein the set ofcorrelation elements is indicative of at least an extent of spatialcorrelation between a plurality of residual elements in the set ofresidual elements, and wherein the input data includes data identifyingwhether the set of correlation elements is indicative of the extent ofspatial correlation or whether the set of correlation elements isfurther indicative of an extent of temporal correlation between firstreference data based on the first rendition and second reference databased on a rendition of a second time sample of the signal.

Further features and advantages will become apparent from the followingdescription, given by way of example only, which is made with referenceto the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic block diagram of an example of a signalprocessing system in accordance with an embodiment of the presentinvention;

FIGS. 2A and 2B show a schematic block diagram of another example of asignal processing system in accordance with an embodiment of the presentinvention;

FIGS. 3A and 3B show a schematic block diagram of another example of asignal processing system in accordance with an embodiment of the presentinvention;

FIGS. 4A and 4B show a schematic block diagram of another example of asignal processing system in accordance with an embodiment of the presentinvention;

FIGS. 5A and 5B show a schematic block diagram of another example of asignal processing system in accordance with an embodiment of the presentinvention;

FIG. 6 is a flow diagram depicting an example of a method in accordancewith an embodiment of the present invention;

FIG. 7 is a flow diagram depicting another example of a method inaccordance with an embodiment of the present invention; and

FIG. 8 shows a schematic block diagram of an example of an apparatus inaccordance with an embodiment of the present invention.

DETAILED DESCRIPTION

Referring to FIG. 1, there is shown an example of a signal processingsystem 100. The signal processing system 100 is used to process signals.Examples of types of signal include, but are not limited to, videosignals, image signals, audio signals, volumetric signals such as thoseused in medical, scientific or holographic imaging, or othermultidimensional signals.

The signal processing system 100 includes a first apparatus 102 and asecond apparatus 104. The first apparatus 102 and second apparatus 104may have a client-server relationship, with the first apparatus 102performing the functions of a server device and the second apparatus 104performing the functions of a client device. The signal processingsystem 100 may include at least one additional apparatus (not shown).The first apparatus 102 and/or second apparatus 104 may comprise one ormore components. The components may be implemented in hardware and/orsoftware. The one or more components may be co-located or may be locatedremotely from each other in the signal processing system 100. Examplesof types of apparatus include, but are not limited to, computeriseddevices, routers, workstations, handheld or laptop computers, tablets,mobile devices, games consoles, smart televisions, set-top boxes,augmented and/or virtual reality headsets etc.

The first apparatus 102 is communicatively coupled to the secondapparatus 104 via a data communications network 106. Examples of thedata communications network 106 include, but are not limited to, theInternet, a Local Area Network (LAN) and a Wide Area Network (WAN). Thefirst and/or second apparatus 102, 104 may have a wired and/or wirelessconnection to the data communications network 106.

The first apparatus 102 comprises an encoder device 108. The encoderdevice 108 is configured to encode data comprised in the signal, whichis referred to hereinafter as “signal data”. The encoder device 108 mayperform one or more further functions in addition to encoding signaldata. The encoder device 108 may be embodied in various different ways.For example, the encoder device 108 may be embodied in hardware and/orsoftware.

The second apparatus 104 comprises a decoder device 110. The decoderdevice 110 is configured to decode signal data. The decoder device 110may perform one or more further functions in addition to decoding signaldata. The decoder device 110 may be embodied in various different ways.For example, the decoder device 110 may be embodied in hardware and/orsoftware.

The encoder device 108 encodes signal data and transmits the encodedsignal data to the decoder device 110 via the data communicationsnetwork 106. The decoder device 110 decodes the received, encoded signaldata and generates decoded signal data. The decoder device 110 mayoutput the decoded signal data, or data derived using the decoded signaldata. For example, the decoder device 110 may output such data fordisplay on one or more display devices associated with the secondapparatus 104.

In some examples described herein, the encoder device 108 transmits tothe decoder device 110 a rendition of a signal at a given level ofquality and information the decoder device 110 can use to reconstruct arendition of the signal at one or more higher levels of quality. Arendition of a signal at a given level of quality may be considered tobe a representation, version or depiction of data comprised in thesignal at the given level of quality.

Compared to some known techniques, examples described herein allow arelatively small amount of information to be used for suchreconstruction. This may reduce the amount of data transmitted via thedata communications network 106. The savings may be particularlyrelevant where the signal data corresponds to high quality video data,where the amount of information transmitted in known systems can beespecially high.

Referring to FIGS. 2A and 2B, there is shown schematically an example ofa signal processing system 200. The signal processing system 200includes a first apparatus 202 comprising an encoder device and a secondapparatus 204 comprising a decoder device. In each of the firstapparatus 202 and the second apparatus 204, items are shown on twological levels. The two levels are separated by a dashed line. Items onthe first, highest level relate to data at a relatively high level ofquality. Items on the second, lowest level relate to data at arelatively low level of quality. The relatively high and relatively lowlevels of quality relate to a tiered hierarchy having multiple levels ofquality. In some examples, the tiered hierarchy comprises more than twolevels of quality. In such examples, the first apparatus 202 and thesecond apparatus 204 may include more than two different levels. Theremay be one or more other levels above and/or below those depicted inFIGS. 2A and 2B.

Referring first to FIG. 2A, the first apparatus 202 obtains a renditionof a first time sample, t₁, of a signal at a relatively high level ofquality 206. A rendition of a given time sample of a signal is arepresentation of data comprised in the signal at a given point in time.The rendition of the first time sample, t₁, of the signal at therelatively high level of quality 206 will be referred to as “input data”hereinafter as, in this example, it is data provided as an input to theencoder device in the first apparatus 202. The first apparatus 202 mayreceive the input data 206. For example, the first apparatus 202 mayreceive the input data 206 from at least one other apparatus.

When the signal is a video signal, the first time sample of the signalmay be all or part of an image or frame from a sequence of images orframes making up the video signal. The input data 206 may be arranged asan array of signal elements, I_(ij), comprising one or more rows andcolumns of signal elements, where i denotes a row number associated withthe signal element and j denotes a column number associated with thesignal element. In this example, the input data 206 is arranged as anarray comprising first and second rows of signal elements. The first rowincludes signal elements I₁₁ and I₁₂ and the second row includes signalelements I₂₁ and I₂₂. In this example, the input data 206 relates topart of an image. The part of the image may be referred to as a tile.The entire image may comprise many such tiles, and may consequentlycomprise many more than four signal elements. However, for convenienceand brevity, in this example that the input data comprises just foursignal elements.

The first apparatus 202 derives data 212 based on the input data 206. Inthis example, the data 212 based on the input data 206 is a preliminaryrepresentation 212 of the first time sample, t₁, of the signal at therelatively low level of quality. In this example, the data 212 isderived by performing a downsampling operation on the input data 206 andwill therefore be referred to as “downsampled data” hereinafter. Inother examples, the data 212 is derived by performing an operation otherthan a downsampling operation on the input data 206.

In this example, the downsampled data 212 is processed to generateprocessed data 213 at the relatively low level of quality. In otherexamples, the downsampled data 212 is not processed at the relativelylow level of quality. As such, the first apparatus 202 may generate dataat the relatively low level of quality, where the data at the relativelylow level of quality comprises the downsampled data 212 or the processeddata 213.

In some examples, generating the processed data 213 involves encodingthe downsampled data 212. Encoding the downsampled data 212 produces anencoded signal at the relatively low level of quality. The firstapparatus 202 may output the encoded signal, for example fortransmission to the second apparatus 204. Instead of being produced inthe first apparatus 202, the encoded signal may be produced by anencoding device that is separate from the first apparatus 202. Theencoded signal may be an H.264 encoded signal. H.264 encoding caninvolve arranging a sequence of images into a Group of Pictures (GOP).Each image in the GOP is representative of a different time sample ofthe signal. A given image in the GOP may be encoded using one or morereference images associated with earlier and/or later time samples fromthe same GOP, in a process known as ‘inter-frame prediction’.

Generating the processed data 213 at the relatively low level of qualitymay further involve decoding the encoded signal at the relatively lowlevel of quality. The decoding operation may be performed to emulate adecoding operation at the second apparatus 204, as will become apparentbelow. Decoding the encoded signal produces a decoded signal at therelatively low level of quality. In some examples, the first apparatus202 decodes the encoded signal at the relatively low level of quality toproduce the decoded signal at the relatively low level of quality. Inother examples, the first apparatus 202 receives the decoded signal atthe relatively low level of quality, for example from an encoding and/ordecoding device that is separate from the first apparatus 202. Theencoded signal may be decoded using an H.264 decoder. H.264 decodingresults in a sequence of images (that is, a sequence of time samples ofthe signal) at the relatively low level of quality. None of theindividual images is indicative of a temporal correlation betweendifferent images in the sequence following the completion of the H.264decoding process. Therefore, any exploitation of temporal correlationbetween sequential images that is employed by H.264 encoding is removedduring H.264 decoding, as sequential images are decoupled from oneanother. The processing that follows is therefore performed on animage-by-image basis where the first apparatus 202 processes videosignal data.

In an example, generating the processed data 213 at the relatively lowlevel of quality further involves obtaining correction data based on acomparison between the downsampled data 212 and the decoded signalobtained by the first apparatus 202, for example based on the differencebetween the downsampled data 212 and the decoded signal. The correctiondata can be used to correct for errors introduced in encoding anddecoding the downsampled data 212. In some examples, the first apparatus202 outputs the correction data, for example for transmission to thesecond apparatus 204, as well as the encoded signal. This allows therecipient to correct for the errors introduced in encoding and decodingthe downsampled data 212.

In some examples, generating the processed data 213 at the relativelylow level of quality further involves correcting the decoded signalusing the correction data. In other examples, rather than correcting thedecoded signal using the correction data, the first apparatus 202 usesthe downsampled data 212.

In some examples, generating the processed data 213 involves performingone or more operations other than the encoding, decoding, obtaining andcorrecting acts described above.

The first apparatus 202 obtains data 214 based on the data at therelatively low level of quality. As indicated above, the data at therelatively low level of quality may comprise the processed data 213, orthe downsampled data 212 where the downsampled data 212 is not processedat the lower level. In this example, the data 214 is a second renditionof the first time sample of the signal at the relatively high level ofquality, the first rendition of the first time sample of the signal atthe relatively high level of quality being the input data 206. Thesecond rendition at the relatively high level of quality may beconsidered to be a preliminary or predicted rendition of the first timesample of the signal at the relatively high level of quality. In thisexample, the first apparatus 202 derives the data 214 by performing anupsampling operation on the data at the relatively low level of quality.The data 214 will be referred to hereinafter as “upsampled data”.However, in other examples one or more other operations could be used toderive the data 214, for example where data 212 is not derived bydownsampling the input data 206.

Similar to the input data 206, the upsampled data 214 may be arranged asan array of signal elements, U_(ij), comprising one or more rows andcolumns of signal elements, where i denotes a row number associated withthe signal element and j denotes a column number associated with thesignal element.

The input data 206 and the upsampled data 214 are used to obtainresidual data 216. The residual data 216 is associated with the firsttime sample, t₁, of the signal. The residual data 216 may be in the formof a set of residual elements. In this example, the set of residualelements 216 is arranged as an array of residual elements comprisingfirst and second rows of residual elements. The first row includesresidual elements r₁₁ and r₁₂ and the second row includes residualelements r₂₁ and r₂₂. A residual element in the set of residual elements216 is associated with a respective signal element in the input data206. For example, residual element r₁₂ in the residual data 216 isassociated with respective signal element I₁₂ in the input data 206.

In this example, a given residual element is obtained by subtracting avalue of a signal element in the upsampled data 214 from a value of acorresponding signal element in the input data 206. As such,r_(ij)=I_(ij)−U_(ij), where i indicates a row number associated with theelement and j indicates a column number associated with the element.Since I_(ij)=U_(ij)+r_(ij), the set of residual elements 216 is useablein combination with the upsampled data 214 to reconstruct the input data206. The residual data 216 may also be referred to as “reconstructiondata” or “enhancement data”.

A set of spatial correlation elements 218 is generated using the set ofresidual elements 216. The term “spatial correlation element” is usedherein to indicate an element that is indicative of an extent, ormeasure, of spatial correlation between a plurality of residual elementsin the set of residual elements 216. The correlation elements in the setof spatial correlation elements 218 may also be referred to as“coefficients”, “spatial coefficients” or “transformed elements”. Theset of spatial correlation elements 218 is associated with the firsttime sample, t₁, of the signal. In this example, the set of spatialcorrelation elements 218 is arranged as an array of data elementscomprising first and second rows of signal elements. In this example,the set of spatial correlation elements 218 is arranged as a 2×2 arrayof correlation elements, A, H, V, D. The elements in the set of spatialcorrelation elements 218 may be arranged in a form other than a 2×2array, for example a 4×1 or 1×4 array.

In this example, the set of spatial correlation elements 218 is derivedby pre-multiplying the set of residual elements 216 with atransformation matrix (or ‘kernel’), K. In this example, thetransformation matrix is a 4×4 matrix:

$K = {{\frac{1}{4}\begin{bmatrix}1 & 1 & 1 & 1 \\1 & {- 1} & 1 & {- 1} \\1 & 1 & {- 1} & {- 1} \\1 & {- 1} & {- 1} & 1\end{bmatrix}}.}$

In this example, the set of spatial correlation elements 218 is derivedbased on the following equation, where the set of spatial correlationelements 218 and the set of residual elements 216 are both shown as 4×1arrays:

$\begin{matrix}{\begin{bmatrix}A \\H \\V \\D\end{bmatrix} = {{\frac{1}{4}\begin{bmatrix}1 & 1 & 1 & 1 \\1 & {- 1} & 1 & {- 1} \\1 & 1 & {- 1} & {- 1} \\1 & {- 1} & {- 1} & 1\end{bmatrix}} \cdot \begin{bmatrix}r_{11} \\r_{12} \\r_{21} \\r_{22}\end{bmatrix}}} & \left( {{eq}.\mspace{14mu} 1} \right)\end{matrix}$

-   -   As such,

${A = {\frac{1}{4}\left( {r_{11} + r_{12} + r_{21} + r_{22}} \right)}},{H = {\frac{1}{4}\left( {r_{11} - r_{12} + r_{21} - r_{22}} \right)}},{V = {\frac{1}{4}\left( {r_{11} + r_{12} - r_{21} - r_{22}} \right)\mspace{14mu}{and}}}$$D = {\frac{1}{4}{\left( {r_{11} - r_{12} - r_{21} + r_{22}} \right).}}$

A represents an average of the residual elements in the set of residualelements 216. H represents a horizontal correlation and/or ‘tilt’between the residual elements in the set of residual elements 216. Vrepresents a vertical correlation and/or ‘tilt’ between the residualelements in the set of residual elements 216. D represents a diagonalcorrelation and/or ‘tilt’ between the residual elements in the set ofresidual elements 216. The correlation elements described herein thusexploit, and are indicative of an extent of, directional and/or spatialcorrelation between neighbouring residual elements.

The set of spatial correlation elements 218 may include a delta averagevalue, Δ_(A), instead of an average value, A. In some examples, Δ_(A) isdefined as the difference between an average of signal elements in theinput data 206 and an average of corresponding signal elements in thedata at the lower level of quality, for example the downsampled data 212or the processed data 213. In some examples, Δ_(A) is derived based on:the average of the values in the set of residual elements 216, A, anaverage of signal elements in the data at the lower level of quality andan average of signal elements in the upsampled data 214. Δ_(A) may besmaller than A and/or may be more likely to be zero than A. This mayfacilitate efficient encoding of the set of spatial correlation elements218 and a reduced amount of data transfer between the first apparatus202 and the second apparatus 204 compared to using the average value, A.The delta average value, Δ_(A), may be provided to the second apparatus202 to enable the second apparatus to calculate the average value, A.

Instead of the first apparatus 202 transmitting the set of residualelements 216 to the second apparatus 204, in some examples the firstapparatus 202 transmits the set of spatial correlation elements 218instead. Since the set of spatial correlation elements 218 exploitspatial redundancy between a plurality of residual elements, the set ofspatial correlation elements 218 are likely to be smaller than the setof residual elements 216, for example when there is a relatively strongextent of spatial correlation between residual elements in the set ofresidual elements 216 at the higher, residual level. The set of spatialcorrelation elements 218 may comprise spatial correlation elementshaving values of zero in some cases, which may be particularly efficientto encode. Less data may therefore be used to transmit the set ofspatial correlation elements 218 than the set of residual elements 216,for example when there is a relatively strong extent of spatialcorrelation between residual elements in the set of residual elements216. The amount of data needed to transmit the set of spatialcorrelation elements 218 may be larger than or the same size as theamount of data needed to transmit the set of residual elements 216, forexample when there is a relatively weak amount of spatial correlation atthe residual level.

In this example, the first apparatus 202 transmits output data based onthe downsampled data 212 and also transmits the set of spatialcorrelation elements 218 to the second apparatus 204.

Turning now to FIG. 2B, the second apparatus 204 receives data 220 basedon the downsampled data 212 and also receives the set of spatialcorrelation elements 218. The data 220 based on the downsampled data 212may be the downsampled data 212 itself, the processed data 213, or dataderived from the downsampled data 212 or the processed data 213.

In some examples, the received data 220 comprises the processed data213, which may comprise the encoded signal at the relatively low levelof quality and/or the correction data. In some examples, for examplewhere the first apparatus 202 has processed the downsampled data 212 togenerate the processed data 213, the second apparatus 204 processes thereceived data 220 to generate processed data 222. Such processing by thesecond apparatus 204 may comprise decoding an encoded signal to producea decoded signal at the relatively low level of quality. In someexamples, the processing by the second apparatus 204 comprisescorrecting the decoded signal using obtained correction data. In someexamples, the encoded signal at the relatively low level of quality isdecoded by a decoding device that is separate from the second apparatus204. The encoded signal at the relatively low level of quality may bedecoded using an H.264 decoder.

In other examples, the received data 220 comprises the downsampled data212 and does not comprise the processed data 213. In some such examples,the second apparatus 204 does not process the received data 220 togenerate processed data 222.

The second apparatus 204 uses data at the relatively low level ofquality to derive the upsampled data 214. As indicated above, the dataat the relatively low level of quality may comprise the processed data222, or the received data 220 where the second apparatus 204 does notprocess the received data 220 at the relatively low level of quality.The upsampled data 214 is a preliminary rendition of the first timesample, t₁, of the signal at the relatively high level of quality. Theupsampled data 214 may be derived by performing an upsampling operationon the data at the relatively low level of quality.

The second apparatus 204 obtains the set of residual elements 216 basedat least in part on the set of spatial correlation elements 218 receivedfrom the first apparatus 202. The set of residual elements 216 isuseable with the upsampled data 214 to reconstruct the input data 206.The set of residual elements 216 is indicative of a comparison betweenthe input data 206 and the upsampled data 214.

The second apparatus 204 may retrieve an inverse of the transformationmatrix used by the first apparatus 202 to generate the set of spatialcorrelation elements 218 and derive the set of residual elements 216based on a pre-multiplication of the set of spatial correlation elements218 by the inverse of the transformation matrix.

The set of spatial correlation elements 218 exploit spatial correlationbetween residual elements. However, since the set of residual elements216, and therefore also the set of spatial correlation elements 218, areindicative of only a single time sample of the signal, no temporalcorrelation is exploited by the set of spatial correlation elements 218.For example, where the signal is a video signal, the set of residualelements 216 are obtained on an image-by-image basis and do not exploittemporal correlation between the component images of the video signal.

Referring to FIGS. 3A and 3B there is shown schematically an example ofa signal processing system 300. Some items depicted in FIGS. 3A and 3Bare similar to items shown in FIGS. 2A and 2B. Corresponding referencesigns, incremented by 100, have therefore been used for similar items.

Referring first to FIG. 3A, the first apparatus 302 obtains input data306 at the relatively high level of quality. The input data 306comprises a first rendition of a first time sample, t₁, of a signal atthe relatively high level of quality. The first apparatus 302 uses theinput data 306 to derive downsampled data 312 at the relatively lowlevel of quality, for example by performing a downsampling operation onthe input data 306. Where the downsampled data 312 is processed at therelatively low level of quality, such processing generates processeddata 313 at the relatively low level of quality. However, as indicatedabove, in some examples no processing is performed on the downsampleddata 312. Data at the relatively low level of quality is used to deriveupsampled data 314 at the relatively high level of quality, for exampleby performing an upsampling operation on the data at the relatively lowlevel of quality. The upsampled data 314 comprises a second rendition ofthe first time sample of the signal at the relatively high level ofquality. The first apparatus 302 obtains a set of residual elements 316useable to reconstruct the input data 306 using the upsampled data 314.The set of residual elements 316 is associated with the first timesample, t₁, of the signal. The set of residual elements 316 is obtainedby comparing the input data 306 with the upsampled data 314.

In this example, the first apparatus 302 generates a set ofspatio-temporal correlation elements 326. The term “spatio-temporalcorrelation element” is used herein to mean a correlation element thatindicates, in addition to an extent of spatial correlation betweenresidual elements, an extent of temporal correlation. In this example,the set of spatio-temporal correlation elements 326 is associated withboth the first time sample, t₁, of the signal, and a second time sample,t₀, of the signal. This is in contrast with the spatial correlationelements in the set of spatial correlation elements 218 described above,which are associated with only the first time sample of the signal, t₁.In the examples described herein, the second time sample, t₀, is anearlier time sample relative to the first time sample. In otherexamples, however, the second time sample, t₀, is a later time samplerelative to the first time sample, t₁. In some examples, where the inputdata 306 comprises a sequence of time samples, an earlier time samplemeans a time sample that precedes the first time sample, t₁, in theinput data. Where the first time sample, t₁, and the earlier time sampleare arranged in presentation order, the earlier time sample precedes thefirst time sample, t₁.

The second time sample, t₀, may be an immediately preceding time samplein relation to the first time sample, t₁. In some examples, the secondtime sample, t₀, is a preceding time sample relative to the first timesample, t₁, but not an immediately preceding time sample relative to thefirst time sample, t₁.

In this example, the set of spatio-temporal correlation elements 326 isindicative of an extent of spatial correlation between a plurality ofresidual elements in the set of residual elements 316. The set ofspatio-temporal correlation elements 326 is also indicative of an extentof temporal correlation between first reference data based on the inputdata 306 and second reference data based on a rendition of the secondtime sample, t₀, of the signal, for example at the relatively high levelof quality. The first reference data is therefore associated with thefirst time sample, t₁, of the signal, and the second reference data isassociated with the second time sample, t₀, of the signal. The firstreference data and the second reference data are used as references orcomparators for determining an extent of temporal correlation inrelation to the first time sample, t₁, of the signal and the second timesample, t₂, of the signal. The first reference data and/or the secondreference data may be at the relatively high level of quality.

In some examples, the first reference data and the second reference datacomprise first and second sets of spatial correlation elements,respectively, the first set of spatial correlation elements beingassociated with the first time sample, t₁, of the signal, and the secondset of spatial correlation elements being associated with the secondtime sample, t₀, of the signal.

In other examples, the first reference data and the second referencedata comprise first and second renditions of the signal, respectively,the first rendition being associated with the first time sample, t₁, ofthe signal, and the second rendition being associated with the secondtime sample, t₀, of the signal.

The set of spatio-temporal correlation elements 326 will be referred tohereinafter as “Δ_(t) correlation elements”, since, in addition toexploiting the spatial correlation between residual elements, temporalcorrelation is also exploited using data from a different time sample togenerate the Δ_(t) correlation elements 326.

In this example, instead of the first apparatus 302 transmitting the setof spatial correlation elements 218 to the second apparatus 304, thefirst apparatus 302 transmits the set of Δ_(t) correlation elements 326instead. Since the set of Δ_(t) correlation elements 326 exploittemporal redundancy at the higher, residual level, the set of Δ_(t)correlation elements 326 are likely to be smaller than the set ofspatial correlation elements 218, for example where there is a strongtemporal correlation, and may comprise more correlation elements withzero values in some cases. Less data may therefore be used to transmitthe set of Δ_(t) correlation elements 326 than the set of spatialcorrelation elements 218.

Turning now to FIG. 3B, the second apparatus 304 receives data 320 basedon the downsampled data 312 and receives the set of Δ_(t) correlationelements 326.

Where the first apparatus 302 has processed the downsampled data 312 togenerate processed data 313, the second apparatus 304 processes thereceived data 320 to generate processed data 322. The processing maycomprise decoding an encoded signal to produce a decoded signal at therelatively low level of quality. As indicated above, in some examples,the second apparatus 304 does not perform such processing on thereceived data 320. Data at the relatively low level of quality, forexample the received data 320 or the processed data 322, is used toderive the upsampled data 314. The upsampled data 314 may be derived byperforming an upsampling operation on the data at the relatively lowlevel of quality.

The second apparatus 304 obtains the set of residual elements 316 basedat least in part on the set of Δ_(t) correlation elements 326. The setof residual elements 316 is useable to reconstruct the input data 306using the upsampled data 314.

Referring to FIGS. 4A and 4B there is shown schematically an example ofa signal processing system 400. Some items depicted in FIGS. 4A and 4Bare similar to items shown in FIGS. 2A and 2B. Corresponding referencesigns, incremented by 200, have therefore been used for similar items.

Referring first to FIG. 4A, the first apparatus 402 obtains input data406. The input data 406 comprises a first rendition of a first timesample, t₁, of a signal at a relatively high level of quality. The firstapparatus 402 uses the input data 406 to derive downsampled data 412.The downsampled data 412 comprises a rendition of the first time sample,t₁, of the signal at the relatively low level of quality. In someexamples, the downsampled data 412 is processed to generate processeddata 413. In other examples, no processing is performed on thedownsampled data 412. Data at the relatively low level of quality, forexample downsampled data 412 or processed data 413, is used to deriveupsampled data 414. The upsampled data 414 comprises a second renditionof the first time sample, t₁, of the signal at the relatively high levelof quality. The first apparatus 402 obtains a set of residual elements416 by comparing the input data 406 with the upsampled data 414.

In this example, the first apparatus 402 generates a first set ofspatial correlation elements 418 using the set of residual elements 416.The first set of spatial correlation elements 418 is associated with thefirst time sample, t₁, of the signal. At least one correlation elementin the first set of spatial correlation elements 418 is indicative of anextent of spatial correlation between a plurality of residual elementsin the set of residual elements 416. At least one correlation element inthe first set of spatial correlation elements 418 may, for example,indicate a horizontal, vertical and/or diagonal similarity and/or “tilt”between neighbouring residual elements in the set of residual elements416. The first set of spatial correlation elements 418 exploits spatialcorrelation but not temporal correlation at the higher, residual level.

In this example, the first apparatus 402 obtains a second set of spatialcorrelation elements 424. In this example, the second set of spatialcorrelation elements 424 is associated with a second, earlier timesample, t₀, of the signal. The first apparatus 402 may obtain the secondset of spatial correlation elements 424 from a local buffer for example.The second set of spatial correlation elements 424 is an example of thesecond reference data based on the rendition of the second, earlier timesample, t₀, of the signal and in relation to which Δ_(t) correlationelements can indicate a temporal correlation. At least one correlationelement in the second set of spatial correlation elements 424 isindicative of an extent of spatial correlation between a plurality ofresidual elements in a further set of residual elements, the further setof residual elements being associated with the second, earlier timesample, t₀, of the signal. The further set of residual elements isusable to reconstruct a rendition of the second, earlier time sample,t₀, of the signal at the relatively high level of quality using databased on a rendition of the second, earlier time sample, t₀, of thesignal at the relatively low level of quality.

In this example, the first apparatus 402 generates a set of Δ_(t)correlation elements 426 based on the first set of spatial correlationelements 418, and based on the second set of spatial correlationelements 424. The set of Δ_(t) correlation elements 426 may be generatedbased on a comparison, for example a difference, between the first setof spatial correlation elements 418 and the second set of spatialcorrelation elements 424. Consequently, the set of Δ_(t) correlationelements 426 is associated with the first time sample, t₁, of the signaland with the second, earlier time sample, t₀, of the signal. The set ofΔ_(t) correlation elements 426 is indicative of an extent of spatialcorrelation between a plurality of residual elements in the set ofresidual elements 416 and also an extent of temporal correlation betweenfirst reference data in the form of the first set of spatial correlationelements 418 and second reference data in the form of the second set ofspatial correlation elements 424.

In this example, the first apparatus 402 uses the set of Δ_(t)correlation elements 426 to generate first output data, and uses thedownsampled data 412 to generate second output data. In some examples,the second output data comprises the downsampled data 412. Where thefirst apparatus 402 processes the downsampled data 412 to generate anencoded signal, the second output data comprises the encoded signal. Thefirst and the second output data may be output, for example fortransmission to the second apparatus 404.

Turning to FIG. 4B, the second apparatus 404 receives data 420 based onthe downsampled data 412 and receives the set of Δ_(t) correlationelements 426.

Where the first apparatus 402 has processed the downsampled data 412 togenerate the processed data 413, the received data 420 is processed bythe second apparatus 404 to generate processed data 422 at therelatively low level of quality. In some examples, no processing isperformed at the lower level of quality on the received data 420. Dataat the relatively low level of quality, for example the received data420 or the processed data 422, is used to derive the upsampled data 414.

In this example, the second apparatus 404 obtains the second set ofspatial correlation elements 424. The second set of spatial correlationelements 424 may be retrieved from a buffer in the second apparatus 404.For example, the second apparatus 404 may have previously received thesecond set of spatial correlation elements 424 from the first apparatus402, or may have previously derived the second set of spatialcorrelation elements 424 from data received from the first apparatus402.

The second apparatus 404 obtains the first set of spatial correlationelements 418 based on the received set of Δ_(t) correlation elements 426and the obtained second set of spatial correlation elements 424. Thefirst set of spatial correlation elements 418 may be derived bycombining the second set of spatial correlation elements 424 with theset of Δ_(t) correlation elements 426.

The second apparatus 404 derives the set of residual elements 416 usingthe first set of spatial correlation elements 418. The second apparatus404 uses the set of residual elements 416 and the upsampled data 414 toreconstruct the input data 406.

Referring to FIGS. 5A and 5B, there is shown schematically an example ofa signal processing system 500. Some items depicted in FIGS. 5A and 5Bare similar to items shown in FIGS. 2A and 2B. Corresponding referencesigns, incremented by 300, have therefore been used for similar items.

Referring first to FIG. 5A, the first apparatus 502 obtains input data506. The input data 506 comprises a first rendition of a first timesample, t₁, of a signal at a relatively high level of quality.

In this example, the first apparatus 502 obtains a rendition 508 of asecond, earlier time sample, t₀, of the signal at the relatively highlevel of quality. For example, the first apparatus 502 may retrieve therendition 508 of a second, earlier time sample, t₀, of the signal at therelatively high level of quality from a local buffer. In this example,the rendition 508 of the second, earlier time sample, t₀, of the signalis a reconstructed rendition of the second, earlier time sample, t₀, ofthe signal at the relatively high level of quality and will be referredto hereinafter as “reconstructed data”. For example, the reconstructeddata may have been obtained by downsampling and upsampling a renditionof the second, earlier time sample, t₀, of the signal in a similarmanner to that described above. The reconstructed data 508 is anotherexample of second reference data based on a rendition of the second,earlier time sample, t₀, of the signal.

The first apparatus 502 derives a differential rendition 510 of thesignal at the relatively high level of quality based on first referencedata in the form of the input data 506 and second reference data in theform of the reconstructed data 508. The differential rendition 510 maybe derived based on a comparison, for example a difference, between theinput data 506 and the reconstructed data 508. The differentialrendition 510 is hereinafter referred to as “Δ_(t) input data” and isassociated with both the first time sample, t₁, of the signal and thesecond, earlier time sample, t₀, of the signal. The Δ_(t) input data 510is indicative of an extent of temporal correlation between the inputdata 506 and the reconstructed data 508.

In this example, the first apparatus 502 derives a differentialrendition 512 of the signal at the relatively low level of quality basedon the Δ_(t) input data 510. The differential rendition 512 at therelatively low level of quality is referred to herein as “Δ_(t)downsampled data”. In some examples, the Δ_(t) downsampled data 512 isprocessed to generate Δ_(t) processed data 513. In other examples, noprocessing is performed at the relatively low level of quality on theΔ_(t) downsampled data 512. Data at the relatively low level of quality,for example the Δ_(t) downsampled data 512 or the Δ_(t) processed data513, is used to derive a preliminary differential rendition 514 of thesignal at the relatively high level of quality. The preliminarydifferential rendition 514 is referred to hereinafter as “Δ_(t)upsampled data 514”.

In this example, the first apparatus 502 obtains a set of Δ_(t) residualelements 516 by comparing the Δ_(t) input data 510 with the Δ_(t)upsampled data 514. The set of Δ_(t) residual elements 516 is associatedwith both the first time sample, t₁, of the signal and the second,earlier time sample, t₀, of the signal.

In this example, the first apparatus 502 generates a set of Δ_(t)correlation elements 526 using the set of Δ_(t) residual elements 516.The set of Δ_(t) correlation elements 526 is associated with the firsttime sample, t₁, of the signal and the second, earlier time sample, t₀,of the signal. The set of Δ_(t) correlation elements 526 is indicativeof an extent of spatial correlation between a plurality of Δ_(t)residual elements in the set of Δ_(t) residual elements 516 and anextent of temporal correlation between first reference data in the formof the input data 506 and second reference data in the form of thereconstructed data 508.

In this example, the first apparatus 502 generates first output datausing the set of Δ_(t) correlation elements 526 and generates secondoutput data using the Δ_(t) downsampled data 512. The first output dataand second output data may be output, for example for transmission tothe second apparatus 504.

Turning to FIG. 5B, the second apparatus 504 receives Δ_(t) data 520based on the Δ_(t) downsampled data 512 and also receives the set ofΔ_(t) correlation elements 526.

Where the first apparatus 502 has processed the Δ_(t) downsampled data512 at the relatively low level of quality to generate the Δ_(t)processed data 513, the second apparatus 504 processes the receivedΔ_(t) data 520 at the relatively low level of quality to generate Δ_(t)processed data 522. Data at the relatively low level of quality, forexample the received Δ_(t) data 520 or the Δ_(t) processed data 522, isused to derive the Δ_(t) upsampled data 514.

The second apparatus 504 obtains the set of Δ_(t) residual elements 516based at least in part on the set of Δ_(t) correlation elements 526. Theset of Δ_(t) residual elements 516 may be derived by pre-multiplying theset of Δ_(t) correlation elements 526 with an inverse transformationmatrix.

In this example, the second apparatus 502 reconstructs the Δ_(t) inputdata 510 based on the set of Δ_(t) residual elements 516 and the Δ_(t)upsampled data 514. The Δ_(t) input data 510 may be derived by combiningthe set of Δ_(t) residual elements 516 with the Δ_(t) upsampled data514.

In this example, the second apparatus 504 obtains the reconstructed data508. The reconstructed data 508 may, for example, be retrieved from abuffer at the second apparatus 504. The second apparatus 504 may havederived the reconstructed data 508 previously.

The second apparatus 504 reconstructs the input data 506 based on the Ainput data 510 and the reconstructed data 508. The input data 506comprises a rendition of the first time sample, t₁, of the signal at therelatively high level of quality.

Referring to FIG. 6, there is shown an example of a method 600 ofprocessing data. The method 600 may be performed by an apparatuscomprising an encoder device such as any of first apparatuses 102, 202,302, 402, 502 described above.

At item 610, a set of spatial correlation elements associated with thefirst time sample and a set of Δ_(t) correlation elements are derived.The set of spatial correlation elements and the set of Δ_(t) correlationelements may both be derived so that they can be analysed to determinewhether use of the set of spatial correlation elements or the set ofΔ_(t) correlation elements would be preferable. This may be based on theamount of data that would be used to transmit the set of spatialcorrelation elements or the set of Δ_(t) correlation elements.

At item 620, the derived set of spatial correlation elements and thederived set of A correlation elements are used to perform a comparisonbetween the derived set of spatial correlation elements and the derivedset of Δ_(t) correlation elements. The comparison may, for example,comprise a rate-distortion analysis conducted in relation to the set ofspatial correlation elements and the set of Δ_(t) correlation elements.In such a rate-distortion analysis, the “rate” may be indicative of acalculated number of bits per data symbol to be stored and/ortransmitted, and the “distortion” may be indicative of an estimatederror, for example a mean-squared error, arising from the reconstructionof the data by a receiver. The set of spatial correlation elements andthe derived set of Δ_(t) correlation elements may be compared in orderto reduce the number of bits to be transmitted while not exceeding apredetermined amount of distortion. In this example, the comparisoncomprises determining whether a sum of absolute values of the set ofΔ_(t) correlation elements is less than a sum of absolute values of theset of spatial correlation elements.

In some examples, the comparison comprises generating firstreconstructed data and second reconstructed data. The firstreconstructed data is generated using the set of A correlation elements.The second reconstructed data is generated using the set of spatialcorrelation elements. The first reconstructed data and the secondreconstructed data may each be compared to input data. For example, afirst sum of absolute differences may be determined between the firstreconstructed data and the input data and a second sum of absolutedifferences may be determined between the second reconstructed data andthe input data. A minimum of the first sum of absolute differences andthe second sum of absolute differences may be determined. In suchexamples, the comparison results in a determination of which one of theset of Δ_(t) correlation elements and the set of spatial correlationelements produces a better reconstruction, that is, a reconstructionthat is a closer representation of the input data, by virtue of ithaving a lower sum of absolute differences than the other.

Quantisation may be performed on the set of spatial correlation elementsand the derived set of Δ_(t) correlation elements before the comparison.Performing quantisation before the comparison may allow quantisationerrors to be known and/or taken into account during the comparison.

At item 630, if it is determined at item 620 that the sum of theabsolute values of the set of Δ_(t) correlation elements is less thanthe sum of the absolute values of the set of spatial correlationelements, the set of Δ_(t) correlation elements is selected forinclusion in the first output data. In such examples, the first outputdata comprises the set of Δ_(t) correlation elements. This case may bereferred to as “intra- and inter-frame” coding, since the first outputdata is indicative of both spatial (“intra-frame”) and temporal(“inter-frame”) correlation.

At item 640, if it is determined at item 620 that the sum of theabsolute values of the set of Δ_(t) correlation elements is not lessthan the sum of the absolute values of the set of spatial correlationelements, the set of spatial correlation elements is selected forinclusion in the output data. In such examples, the first output datacomprises the set of spatial correlation elements. This case may bereferred to as “intra-frame only” coding, since the first output data isindicative of spatial correlation (“intra”), but not temporal (“inter”)correlation.

In some examples, the apparatus may be configured such that, if it isdetermined, at item 620, that the sum of the absolute values of theΔ_(t) correlation elements is equal to or negligibly different from thesum of the absolute values of the spatial correlation elements, the setof spatial correlation elements is selected for inclusion in the firstoutput data. In other examples, the apparatus may be configured suchthat, if it is determined at item 620 that the sum of the absolutevalues of the Δ_(t) correlation elements is equal to or negligiblydifferent from the sum of the absolute values of the spatial correlationelements, the set of Δ_(t) correlation elements is selected forinclusion in the first output data.

The derived set of Δ_(t) correlation elements is therefore used togenerate the first output data. In some examples, using the derived setof Δ_(t) correlation elements to generate first output data comprisesselecting either the derived set of Δ_(t) correlation elements or thederived set of correlation elements for inclusion in the first outputdata. The derived set of Δ_(t) correlation elements may therefore beused to generate the first output data even if the derived set of Δ_(t)correlation elements is not itself comprised in the first output data.

The first output data may be outputted for transmission to at least oneother apparatus. The first output data may comprise a sequence ofvalues, for example a bit sequence. In some examples, the first outputdata comprises a quantised version of whichever of the set of Δ_(t)correlation elements and the set of spatial correlation elements isselected for inclusion in the first output data. In some examples, thefirst output data is generated by performing an encoding operation onwhichever of the set of Δ_(t) correlation elements and the set ofspatial correlation elements is selected for inclusion in the firstoutput data.

In some examples, data output by the apparatus includes data indicativeof a result of the selection. In some examples the data indicative ofthe result of the selection is in the first output data. In someexamples, the data indicative of the result of the selection is anindicator flag to indicate whether the set of spatial correlationelements or the set of Δ_(t) correlation elements has been selected foruse in the output data. In some examples, the indicator flag is a singlebit in a bit sequence. The presence of such a flag enables the at leastone other apparatus, for example a decoder device, to determine how thedata it receives has been derived and/or how it may be used.

In some examples, the indicator flag is included in a bit sequenceassociated with a given data element in the output data. The given dataelement may be subject to more quantisation than other data elements inorder to accommodate the indicator flag without increasing the overallsize of the bit sequence. The given data element may be associated withan H correlation element from the set of spatial correlation elements ora ΔH correlation element from the set of Δ_(t) correlation elements. Inother words, the given data element may be indicative of an extent ofhorizontal correlation or “tilt” between a plurality of residualelements. A given data element may be selected to contain the indicatorflag based on a likelihood of its associated bit sequence having atleast one non-zero value. For example, an H spatial correlation elementmay be more likely to have non-zero values in its associated bitsequence than other spatial correlation elements. A bit sequence with ahigher likelihood of having at least one non-zero value may be lessimpacted by the inclusion of an indicator flag than a bit sequence witha lower likelihood of having at least one non-zero value. A given dataelement may be selected to contain the indicator flag based on adetermined effect of additional quantisation of the given data element.For example, additional quantisation of an H spatial correlation elementmay be less detrimental in terms of visual quality than additionalquantisation of other spatial correlation elements.

The method 600 can be performed for an entire time sample of a signal orfor a part of a time sample of a signal. The method 600 can be performedfor different parts of the same time sample of the signal with differentoutcomes. For example, a first part of a first time sample of a signalmay have residual elements with values r_(1,11)(t₁)=4, r_(1,12)(t₁)=−2,r_(1,21)(t₁)=12 and r_(1,22)(t₁)=4. Referring to eq. 1 above, the set ofspatial correlation elements for this first part of the first timesample of the signal, [A₁(t₁), H₁(t₁), V₁(t₁), D₁(t₁)], may thus becalculated as the following: A₁(t₁)=(4−2+12+4)/4=4.5;H₁(t₁)=(4−(−2)+12−4)/4=3.5; V₁(t₁)=(4−2−12−4)/4=−3.5;D₁(t₁)=(4−(−2)−12+4)/4=−0.5. The sum of the absolute values of the setof spatial correlation elements for the first part of the first timesample of signal is 4.5+3.5+3.5+0.5=12.

A second set of spatial correlation elements for this first part of thesignal, associated with an earlier time sample, may have values [A₁(t₀),H₁(t₀), V₁(t₀), D₁(t₀)]=[3.5, 3.5, −1, 0]. Consequently, a derived setof Δ_(t) correlation elements for the first part of the signal may havevalues [ΔA₁(t₁−t₀), ΔH₁(t₁−t₀), ΔV₁(t₁−t₀), ΔD₁(t₁−t₀)]=[4.5−3.5,3.5−3.5, −3.5−(−1), −0.5−0]=[1, 0, −2.5, −0.5]. The sum of the absolutevalues of the set of Δ_(t) correlation elements for the first part ofthe signal is 1+0+2.5+0.5=4. Therefore, for this first part of thesignal, the sum of the absolute values of the set of Δ_(t) correlationelements is less than the sum of the absolute values of the set ofspatial correlation elements. Consequently, the set of Δ_(t) correlationelements may be selected for inclusion in the first output data for thefirst part of the signal, instead of the set of spatial correlationelements being used.

A second part of the first time sample of the signal may, for example,have residual elements with values r_(2,11)(t₁)=2, r_(2,12)(t₁)=6,r_(2,21)(t₁)=−1 and r_(2,22)(t₁)=0. The set of spatial correlationelements for this second part of the given time sample of the signal,[A₂(t₁), H₂(t₁), V₂(t₁), D₂(t₁)], may thus be calculated as thefollowing: A₂(t₁)=(2+6−1+0)/4=1.75; H₂(t₁)=(2−6+(−1)−0)/4=−1.25;V₂(t₁)=(2+6−(−1)−0)/4=2.25; D₂(t₁)=(2−6−(−1)+0)/4=−0.75. The sum of theabsolute values of the set of spatial correlation elements for thesecond part of the first time sample of the signal is1.75+1.25+2.25+0.75=6.

A second set of spatial correlation elements for this second part of thesignal, associated with the earlier time sample, may have values[A₂(t₀), H₂(t₀), V₂(t₀), D₂(t₀)]=[−2.5, 2, 0, 1.5]. Consequently, aderived set of Δ_(t) correlation elements for the second part of thesignal may have values [ΔA₂(t₁−t₀), ΔH₂(t₁−t₀), ΔV₂(t₁−t₀),ΔD₂(t₁−t₀)]=[1.75+2.5), −1.25−2, 2.25−0, −0.75−1.5]=[4.25, −3.25, 2.25,−2.25]. The sum of the absolute values of the set of Δ_(t) correlationelements for the second part of the signal is 4.25+3.25+2.25+2.25=12.Therefore, for this second part of the signal, the sum of the absolutevalues of the set of Δ_(t) correlation elements is not less than the sumof the absolute values of the set of spatial correlation elements.Consequently, the set of spatial correlation elements may be selectedfor inclusion in the first output data for the second part of thesignal, instead of the set of Δ_(t) correlation elements being used.

An indicator flag may be used for each part of the time sample of thesignal to indicate whether set of Δ_(t) correlation elements or the setof spatial correlation elements has been used for that part of thesignal.

Choosing between outputting the set of spatial correlation elements orthe set of Δ_(t) correlation elements for a given part or region of asignal provides a flexible mechanism for exploiting spatial and temporalredundancy at a residual level. For parts of a signal exhibiting arelatively strong degree of temporal correlation at the residual level,Δ_(t) correlation elements are likely to be smaller than spatialcorrelation elements and may comprise more zero values. Less data maytherefore be used to transmit the set of Δ_(t) correlation elements. Onthe other hand, for parts of the signal exhibiting a relatively weakdegree of temporal correlation at the residual level, spatialcorrelation elements may be smaller than Δ_(t) correlation elements andmay comprise more zero values. Less data may therefore be used totransmit the set of spatial correlation elements. Δ_(t) correlationelements may however be more computationally complex to process comparedwith spatial correlation elements, for example by a decoder. An increasein complexity may be weighed against a reduction in an amount of datatraffic in order to intelligently determine whether to use spatialcorrelation elements or Δ_(t) correlation elements for a given part of asignal. Consequently, a flexible and adaptable data processing techniqueis provided. For example, the data processing technique may be adaptedbased on the strength of spatial and temporal correlations at theresidual level.

Referring to FIG. 7, there is shown an example of a method 700 ofprocessing data. The method 700 may be performed by an apparatuscomprising an encoder device such as any of first apparatuses 102, 202,302, 402, 502 described above.

At item 710, first buffer data is retrieved from a buffer. The firstbuffer data may comprise data based on a rendition of an earlier timesample, to of a signal. The data based on a rendition of an earlier timesample, to of a signal may for example be reconstruction data or a setof spatial correlation elements associated with the earlier time sample,to of the signal. In some examples, the second reference data, inrelation to which the set of Δ_(t) correlation elements indicates atemporal correlation, is obtained based on the first buffer data. Thefirst buffer data may comprise and/or may be used to derive, forexample, a set of spatial correlation elements associated with theearlier time sample, t₀, of the signal.

At item 720, a set of spatial correlation elements associated with acurrent time sample, t₁, of the signal, and a set of Δ_(t) correlationelements associated with both the earlier and the current time sample ofthe signal, are generated.

At item 730, it is determined whether to select the set of Δ_(t)correlation elements for inclusion in the first output data, instead ofthe derived set of spatial correlation elements associated with thecurrent time sample, t₁, of the signal.

Second buffer data is generated using the set of Δ_(t) correlationelements and/or the set of special correlation elements associated withthe current time sample of the signal. The second buffer data is used tooverwrite the first buffer data, thereby updating the buffer.

If the set of Δ_(t) correlation elements is selected for inclusion inthe first output data, the second buffer data is generated using the setof Δ_(t) correlation elements at item 740. The buffer may then beupdated. Updating the buffer comprises overwriting the first buffer datawith the second buffer data. In this example, the second buffer data isgenerated by combining the set of Δ_(t) correlation elements with thefirst buffer data. In other words, the set of Δ_(t) correlation elementsare added to the previous buffer contents to generate the new buffercontents.

If the set of spatial correlation elements associated with the currenttime sample of the signal is selected for inclusion in the first outputdata, the second buffer data is generated using the set of spatialcorrelation elements associated with the current time sample of thesignal at item 750. The buffer may then be updated by overwriting thefirst buffer data with the second buffer data. In this example, thesecond buffer data is not derived based on the first buffer data. Inother words, the set of spatial correlation elements replaces theprevious buffer contents instead of being added to it.

Updating the buffer by adding the set of Δ_(t) correlation elements tothe previous buffer contents may, over time, result in the propagationof errors, for example arising from errors associated with quantisation.In some examples, the buffer is updated by overwriting the first bufferdata with second buffer data comprising at least one zero value. Thismay correspond to a ‘reset’ of all or part of the buffer. Consequently,for a later time sample, t₂, of the signal, the first buffer dataretrieved from the buffer is zero. Since the set of Δ_(t) correlationelements results from a difference between the set of correlationelements associated with the later time sample, t₂, of the signal andthe retrieved first buffer data, the set of Δ_(t) correlation elementsand the set of spatial correlation elements associated with the latertime sample, t₂, of the signal will be the same. This allows the bufferto be overwritten with the set of correlation elements associated withthe later time sample, t₂, of the signal, namely the spatial correlationelements, regardless of whether the set of Δ_(t) correlation elements orthe set of spatial correlation elements is selected for inclusion in thefirst output data. Consequently, the propagation of errors that may haveaccumulated from previous time samples can be reduced. In an example,the buffer is reset in this way intermittently, for exampleperiodically. In another example, the buffer is reset when it isdetermined that errors associated with the first buffer data havereached or exceeded a given threshold.

Referring to FIG. 8, there is shown a schematic block diagram of anexample of an apparatus 800.

In an example, the apparatus 800 comprises a decoder device. In anotherexample, the apparatus 800 comprises an encoder device.

Other examples of apparatus 800 include, but are not limited to, amobile computer, a personal computer system, a wireless device, basestation, phone device, desktop computer, laptop, notebook, netbookcomputer, mainframe computer system, handheld computer, workstation,network computer, application server, storage device, a consumerelectronics device such as a camera, camcorder, mobile device, videogame console, handheld video game device, a peripheral device such as aswitch, modem, router, etc., or in general any type of computing orelectronic device.

In this example, the apparatus 800 comprises one or more processors 801configured to process information and/or instructions. The one or moreprocessors 801 may comprise a central processing unit (CPU). The one ormore processors 801 are coupled with a bus 802. Operations performed bythe one or more processors 801 may be carried out by hardware and/orsoftware. The one or more processors 801 may comprise multipleco-located processors or multiple disparately located processors.

In this example, the apparatus 800 comprises computer-useable volatilememory 803 configured to store information and/or instructions for theone or more processors 801. The computer-useable volatile memory 803 iscoupled with the bus 802. The computer-useable volatile memory 803 maycomprise random access memory (RAM).

In this example, the apparatus 800 comprises computer-useablenon-volatile memory 804 configured to store information and/orinstructions for the one or more processors 801. The computer-useablenon-volatile memory 804 is coupled with the bus 802. Thecomputer-useable non-volatile memory 804 may comprise read-only memory(ROM).

In this example, the apparatus 800 comprises one or more data-storageunits 805 configured to store information and/or instructions. The oneor more data-storage units 805 are coupled with the bus 802. The one ormore data-storage units 805 may for example comprise a magnetic oroptical disk and disk drive or a solid-state drive (SSD).

In this example, the apparatus 800 comprises one or more input/output(I/O) devices 806 configured to communicate information to and/or fromthe one or more processors 801. The one or more I/O devices 806 arecoupled with the bus 802. The one or more I/O devices 806 may compriseat least one network interface. The at least one network interface mayenable the apparatus 800 to communicate via one or more datacommunications networks. Examples of data communications networksinclude, but are not limited to, the Internet and a Local Area Network(LAN). The one or more I/O devices 806 may enable a user to provideinput to the apparatus 800 via one or more input devices (not shown).The one or more input devices may include for example a remote control,one or more physical buttons etc. The one or more I/O devices 806 mayenable information to be provided to a user via one or more outputdevices (not shown). The one or more output devices may for exampleinclude a display screen.

Various other entities are depicted for the apparatus 800. For example,when present, an operating system 807, signal processing module 808, oneor more further modules 809, and data 810 are shown as residing in one,or a combination, of the computer-usable volatile memory 803,computer-usable non-volatile memory 804 and the one or more data-storageunits 805. The signal processing module 808 may be implemented by way ofcomputer program code stored in memory locations within thecomputer-usable non-volatile memory 804, computer-readable storage mediawithin the one or more data-storage units 805 and/or other tangiblecomputer-readable storage media. Examples of tangible computer-readablestorage media include, but are not limited to, an optical medium (e.g.,CD-ROM, DVD-ROM or Blu-ray), flash memory card, floppy or hard disk orany other medium capable of storing computer-readable instructions suchas firmware or microcode in at least one ROM or RAM or Programmable ROM(PROM) chips or as an Application Specific Integrated Circuit (ASIC).

The apparatus 800 may therefore comprise a signal processing module 808which can be executed by the one or more processors 801. The signalprocessing module 808 can be configured to include instructions toimplement at least some of the operations described herein. Duringoperation, the one or more processors 801 launch, run, execute,interpret or otherwise perform the instructions in the signal processingmodule 808.

Although at least some aspects of the examples described herein withreference to the drawings comprise computer processes performed inprocessing systems or processors, examples described herein also extendto computer programs, for example computer programs on or in a carrier,adapted for putting the examples into practice. The carrier may be anyentity or device capable of carrying the program.

It will be appreciated that the apparatus 800 may comprise more, fewerand/or different components from those depicted in FIG. 8.

The apparatus 800 may be located in a single location or may bedistributed in multiple locations. Such locations may be local orremote.

The techniques described herein may be implemented in software orhardware, or may be implemented using a combination of software andhardware. They may include configuring an apparatus to carry out and/orsupport any or all of techniques described herein.

The above embodiments are to be understood as illustrative examples.Further embodiments are envisaged.

In examples described above, the first time sample, t₁, and the secondtime sample, t₀, are both time samples of the same signal. In otherexamples, the first time sample, t₁, and the second time sample, t₀, aretime samples of different signals. For example, if the signals are videosignals, the second time sample, t₀, may correspond to a last frame of afirst video and the first time sample, t₁, may correspond to a firstframe of a subsequent video. As such, a set of spatio-temporalcorrelation elements may be indicative of an extent of temporalcorrelation between first reference data based on a first rendition of afirst time sample of a signal and second reference data based on arendition of a second time sample of the or another signal.

In examples described above the second time sample, t₀, is an earliertime sample than the first time sample, t₁. In other examples, thesecond time sample, t₀, is a later time sample than the first timesample, t₁.

In some examples, the decoder device 110 receives input data comprisingfirst input data based on a set of correlation elements and second inputdata based on a rendition of a first time sample of a signal at arelatively low level of quality in a tiered hierarchy having multiplelevels of quality. The decoder device 110 obtains a set of residualelements using the set of correlation elements, the set of residualelements being useable to reconstruct a first rendition of the firsttime sample of the signal at a relatively high level of quality in thetiered hierarchy using a second rendition of the first time sample ofthe signal at the relatively high level of quality. The second renditionis based on the rendition at the relatively low level of quality. Thedecoder device 110 reconstructs the first rendition at the relativelyhigh level of quality using the second rendition and the set of residualelements. The set of correlation elements is indicative of at least anextent of spatial correlation between a plurality of residual elementsin the set of residual elements. The input data includes dataidentifying whether the set of correlation elements is indicative of theextent of spatial correlation or whether the set of correlation elementsis further indicative of an extent of temporal correlation between firstdata based on the first rendition and second data based on a renditionof a second, earlier time sample of the signal. In other words, anidentifier is included in the input data, which the decoder device 110uses to identify whether the set of correlation elements is indicativeof both spatial and temporal correlation at the residual level, or onlyspatial correlation (and not temporal correlation) at the residuallevel.

In examples described above, quantisation is performed on the set ofspatial correlation elements and the set of Δ_(t) correlation elementsprior to comparing the two sets. In other examples, quantisation isperformed after the comparison. In other words, output data may begenerated by performing a quantisation operation on the derived set ofspatial correlation elements or the derived set of Δ_(t) correlationelements. Performing quantisation after the set of spatial correlationelements and the derived set of Δ_(t) correlation elements have beencompared may facilitate a reduction in errors and/or lost informationcompared with a case in which quantisation is performed before thecomparison.

In examples described above, a set of residual elements is obtained, theset of residual elements being useable to reconstruct a first renditionof a first time sample of a signal at a relatively high level of qualityin a tiered hierarchy having multiple levels of quality using a secondrendition of the first time sample of the signal at the relatively highlevel of quality, the second rendition being based on a rendition of thefirst time sample of the signal at a relatively low level of quality inthe tiered hierarchy. In other examples, a set of residual elements isobtained, the set of residual elements being useable to reconstruct arendition of a first time sample of a signal using a rendition of asecond, for example earlier, time sample of the signal. In suchexamples, the set of residual elements is not obtained by downsamplingor upsampling the rendition of the first time sample of the signal.

In examples described above, the data processing apparatus 402 obtains afirst set of spatial correlation elements indicative of an extent ofspatial correlation between a first set of residual elements associatedwith a first time sample of a signal, obtains a second set of spatialcorrelation elements indicative of an extent of spatial correlationbetween a second set of residual elements associated with a second timesample of the signal, and generates a set of spatio-temporal correlationelements indicative of an extent of temporal correlation between thefirst set of spatial correlation elements and the second set of spatialcorrelation elements. As such, a determination of spatial correlationprecedes a determination of temporal correlation. In other examples, thedata processing apparatus 402 obtains a first set of residual elementsassociated with the first time sample of the signal and a second set ofresidual elements associated with the second time sample of the signal,generates a set of temporal correlation elements indicative of an extentof temporal correlation between the first set of residual elements andthe second set of residual elements, and generates a set ofspatio-temporal correlation elements indicative of an extent of spatialcorrelation between correlation elements in the set of temporalcorrelation elements. In such examples, a determination of temporalcorrelation precedes a determination of spatial correlation.

Various measures (for example apparatuses, methods and computerprograms) are provided in which a set of residual elements is obtained.The set of residual elements is useable to reconstruct a first renditionof a first time sample of a signal using a second rendition of the firsttime sample of the signal. The first rendition is at a relatively highlevel of quality in a tiered hierarchy having multiple levels ofquality. The second rendition is at the relatively high level ofquality. The second rendition is based on a rendition of the first timesample of the signal at a relatively low level of quality in the tieredhierarchy. A set of spatio-temporal correlation elements are generated.The set of spatio-temporal correlation elements are associated with thefirst time sample of the signal. The set of spatio-temporal correlationelements are indicative of an extent of spatial correlation between aplurality of residual elements in the set of residual elements. The setof spatio-temporal correlation elements are indicative of an extent oftemporal correlation between first reference data based on the firstrendition and second reference data based on a rendition of a secondtime sample of the signal. The set of spatio-temporal correlationelements are used to generate first output data. The rendition at therelatively low level of quality is used to generate second output data.

In examples described above, the first reference data is at therelatively high level of quality.

In examples described above, the second reference data is at therelatively high level of quality.

In examples described above, the second time sample of the signal is anearlier time sample of the signal relative to the first time sample ofthe signal.

In examples described above, a first set of spatial correlation elementsis generated. The first set of spatial correlation elements areassociated with the first time sample of the signal. The first set ofspatial correlation elements are indicative of the extent of spatialcorrelation between the plurality of residual elements in the set ofresidual elements.

In examples described above, the set of spatio-temporal correlationelements is used to select either the set of spatio-temporal correlationelements or the first set of spatial correlation elements for inclusionin the first output data.

In examples described above, the first reference data comprises thefirst set of spatial correlation elements.

In examples described above, the second reference data comprises asecond set of spatial correlation elements associated with the secondtime sample of the signal. The second set of spatial correlationelements may be indicative of an extent of spatial correlation between aplurality of residual elements in a further set of residual elementsassociated with the second time sample. The further set of residualelements may be usable to reconstruct a rendition of the second timesample of the signal at the relatively high level of quality using databased on a rendition of the second time sample of the signal at therelatively low level of quality.

In examples described above, the first reference data comprises thefirst rendition of the first time sample of the signal.

In examples described above, the second reference data comprises areconstructed rendition of the second time sample of the signal at therelatively high level of quality.

In examples described above, the selecting is performed by comparing theset of spatio-temporal correlation elements with the first set ofspatial correlation elements.

In examples described above, the selecting is performed based on arate-distortion analysis conducted in relation to the set ofspatio-temporal correlation elements and the first set of spatialcorrelation elements.

In examples described above, the first output data includes dataindicative of a result of the selection.

In examples described above, the data indicative of the result of theselection is included in a bit sequence associated with a given dataelement in the first output data. The given data element may beindicative of an extent of horizontal correlation between a plurality ofresidual elements in the set of residual elements.

In examples described above, the set of spatio-temporal correlationelements and the first set of spatial correlation elements arequantised.

In examples described above, first buffer data is retrieved from abuffer. The second reference data is obtained based on the first bufferdata.

In examples described above, second buffer data is generated using theset of spatio-temporal correlation elements. The buffer is updated byoverwriting the first buffer data with the second buffer data.

In examples described above, the second buffer data is generated bycombining the set of spatio-temporal correlation elements with the firstbuffer data.

In examples described above, the buffer is updated by overwriting thefirst buffer data with second buffer data comprising at least one zerovalue.

In examples described above, the second rendition is derived byperforming an upsampling operation on the rendition at the relativelylow level of quality.

In examples described above, the rendition of the first time sample ofthe signal at the relatively low level of quality is derived byperforming a downsampling operation on the first rendition.

In examples described above, the signal is a video signal.

In examples described above, the first output data and the second outputdata are output for transmission to at least one other apparatus via oneor more data communication networks.

Various measures (for example apparatuses, methods and computerprograms) are provided in which input data comprising first input dataand second input data is received. The first input data is based on aset of correlation elements. The second input data is based on arendition of a first time sample of a signal at a relatively low levelof quality in a tiered hierarchy having multiple levels of quality. Aset of residual elements is obtained using the set of correlationelements. The set of residual elements is useable to reconstruct a firstrendition of the first time sample of the signal at a relatively highlevel of quality in the tiered hierarchy using a second rendition of thefirst time sample of the signal at the relatively high level of quality.The second rendition is based on the rendition at the relatively lowlevel of quality. The first rendition at the relatively high level ofquality is reconstructed using the second rendition and the set ofresidual elements. The set of correlation elements is indicative of atleast an extent of spatial correlation between a plurality of residualelements in the set of residual elements. The input data includes dataidentifying whether the set of correlation elements is indicative of theextent of spatial correlation or whether the set of correlation elementsis further indicative of an extent of temporal correlation between firstreference data based on the first rendition and second reference databased on a rendition of a second time sample of the signal.

It is to be understood that any feature described in relation to any oneembodiment may be used alone, or in combination with other featuresdescribed, and may also be used in combination with one or more featuresof any other of the embodiments, or any combination of any other of theembodiments. Furthermore, equivalents and modifications not describedabove may also be employed without departing from the scope of theinvention, which is defined in the accompanying claims.

What is claimed is:
 1. An apparatus configured to: obtain a set ofresidual elements, the set of residual elements being useable toreconstruct a first rendition of a first time sample of a signal at arelatively high level of quality in a tiered hierarchy having multiplelevels of quality using a second rendition of the first time sample ofthe signal at the relatively high level of quality, the second renditionbeing based on a rendition of the first time sample of the signal at arelatively low level of quality in the tiered hierarchy; generate a setof spatio-temporal correlation elements associated with the first timesample of the signal, the set of spatio-temporal correlation elementsbeing indicative of an extent of spatial correlation between a pluralityof residual elements in the set of residual elements and an extent oftemporal correlation between first reference data based on the firstrendition and second reference data based on a rendition of a secondtime sample of the signal; and use the set of spatio-temporalcorrelation elements to generate first output data, wherein therendition at the relatively low level of quality is used to generatesecond output data, and wherein the apparatus is configured to generatea first set of spatial correlation elements associated with the firsttime sample of the signal, the first set of spatial correlation elementsbeing indicative of the extent of spatial correlation between theplurality of residual elements in the set of residual elements.
 2. Anapparatus according to claim 1, wherein the second reference data isderived from a second set of spatial correlation elements associatedwith the second time sample of the signal, the second set of spatialcorrelation elements being indicative of an extent of spatialcorrelation between a plurality of residual elements in a further set ofresidual elements associated with the second time sample, the furtherset of residual elements being usable to reconstruct a rendition of thesecond time sample of the signal at the relatively high level of qualityusing data based on a rendition of the second time sample of the signalat the relatively low level of quality.
 3. An apparatus according toclaim 1, the apparatus comprising a buffer, the apparatus beingconfigured to: retrieve first buffer data from the buffer; and obtainthe second reference data based on the first buffer data.
 4. Anapparatus according to claim 3, the apparatus being configured to:generate second buffer data using the set of spatio-temporal correlationelements; and update the buffer by overwriting the first buffer datawith the second buffer data.
 5. An apparatus according to claim 4, theapparatus being configured to generate the second buffer data bycombining the set of spatio-temporal correlation elements with the firstbuffer data.
 6. An apparatus according to claim 3, the apparatus beingconfigured to update the buffer by overwriting the first buffer datawith second buffer data comprising at least one zero value.
 7. Anapparatus according to claim 1, wherein the rendition of the first timesample of the signal at the relatively low level of quality is derivedby performing a downsampling operation on the first rendition andwherein the second rendition is derived by performing an upsamplingoperation on the rendition at the relatively low level of quality.
 8. Anapparatus configured to: receive input data comprising first input databased on a set of spatio-temporal correlation elements and second inputdata based on a rendition of a first time sample of a signal at arelatively low level of quality in a tiered hierarchy having multiplelevels of quality; obtain a set of residual elements using the set ofspatio-temporal correlation elements, the set of residual elements beinguseable to reconstruct a first rendition of the first time sample of thesignal at a relatively high level of quality in the tiered hierarchyusing a second rendition of the first time sample of the signal at therelatively high level of quality, the second rendition being based onthe rendition at the relatively low level of quality; and reconstructthe first rendition at the relatively high level of quality using thesecond rendition and the set of residual elements, wherein the set ofspatio-temporal correlation elements are indicative of an extent ofspatial correlation between a plurality of residual elements in the setof residual elements and an extent of temporal correlation between firstreference data based on the first rendition and second reference databased on a rendition of a second time sample of the signal, wherein thefirst reference data is derived from a first set of spatial correlationelements associated with the first time sample of the signal, the firstset of spatial correlation elements being indicative of the extent ofspatial correlation between the plurality of residual elements in theset of residual elements.
 9. An apparatus according to claim 8, whereinthe second reference data is derived from a second set of spatialcorrelation elements associated with the second time sample of thesignal, the second set of spatial correlation elements being indicativeof an extent of spatial correlation between a plurality of residualelements in a further set of residual elements associated with thesecond time sample, the further set of residual elements being usable toreconstruct the rendition of the second time sample of the signal at therelatively high level of quality using data based on a rendition of thesecond time sample of the signal at the relatively low level of quality.10. An apparatus according to claim 8, the apparatus comprising abuffer, the apparatus being configured to: retrieve first buffer datafrom the buffer; and reconstruct the first rendition using the firstbuffer data.
 11. An apparatus according to claim 10, wherein the firstbuffer data is derived from the second reference data.
 12. An apparatusaccording to claim 10, the apparatus being configured to: generatesecond buffer data using the set of spatio-temporal correlationelements; and update the buffer by overwriting the first buffer datawith the second buffer data.
 13. An apparatus according to claim 12, theapparatus being configured to generate the second buffer data bycombining the set of spatio-temporal correlation elements with the firstbuffer data.
 14. An apparatus according to claim 8, wherein the secondrendition is derived by performing an upsampling operation on therendition at the relatively low level of quality.
 15. A methodcomprising: obtaining a set of residual elements, the set of residualelements being useable to reconstruct a first rendition of a first timesample of a signal at a relatively high level of quality in a tieredhierarchy having multiple levels of quality using a second rendition ofthe first time sample of the signal at the relatively high level ofquality, the second rendition being based on a rendition of the firsttime sample of the signal at a relatively low level of quality in thetiered hierarchy; generating a set of spatio-temporal correlationelements associated with the first time sample of the signal, the set ofspatio-temporal correlation elements being indicative of an extent ofspatial correlation between a plurality of residual elements in the setof residual elements and an extent of temporal correlation between firstreference data based on the first rendition and second reference databased on a rendition of a second time sample of the signal, whereingenerating the set of spatio-temporal correlation elements includesgenerating a first set of spatial correlation elements associated withthe first time sample of the signal, the first set of spatialcorrelation elements being indicative of the extent of spatialcorrelation between the plurality of residual elements in the set ofresidual elements, and using the set of spatio-temporal correlationelements to generate first output data, wherein the rendition at therelatively low level of quality is used to generate second output data.16. A method according to claim 15, wherein the second reference data isderived from a second set of spatial correlation elements associatedwith the second time sample of the signal, the second set of spatialcorrelation elements being indicative of an extent of spatialcorrelation between a plurality of residual elements in a further set ofresidual elements associated with the second time sample, the furtherset of residual elements being usable to reconstruct a rendition of thesecond time sample of the signal at the relatively high level of qualityusing data based on a rendition of the second time sample of the signalat the relatively low level of quality.
 17. A method according to claim15, wherein the rendition of the first time sample of the signal at therelatively low level of quality is derived by performing a downsamplingoperation on the first rendition and wherein the second rendition isderived by performing an upsampling operation on the rendition at therelatively low level of quality.
 18. A method comprising: receivinginput data comprising first input data based on a set of spatio-temporalcorrelation elements and second input data based on a rendition of afirst time sample of a signal at a relatively low level of quality in atiered hierarchy having multiple levels of quality; obtaining a set ofresidual elements using the set of spatio-temporal correlation elements,the set of residual elements being useable to reconstruct a firstrendition of the first time sample of the signal at a relatively highlevel of quality in the tiered hierarchy using a second rendition of thefirst time sample of the signal at the relatively high level of quality,the second rendition being based on the rendition at the relatively lowlevel of quality; and reconstructing the first rendition at therelatively high level of quality using the second rendition and the setof residual elements, wherein the set of spatio-temporal correlationelements are indicative of an extent of spatial correlation between aplurality of residual elements in the set of residual elements and anextent of temporal correlation between first reference data based on thefirst rendition and second reference data based on a rendition of asecond time sample of the signal, and wherein the first reference datais derived from a first set of spatial correlation elements associatedwith the first time sample of the signal, the first set of spatialcorrelation elements being indicative of the extent of spatialcorrelation between the plurality of residual elements in the set ofresidual elements.
 19. A method according to claim 18, wherein thesecond reference data is derived from a second set of spatialcorrelation elements associated with the second time sample of thesignal, the second set of spatial correlation elements being indicativeof an extent of spatial correlation between a plurality of residualelements in a further set of residual elements associated with thesecond time sample, and wherein the further set of residual elements isusable to reconstruct the rendition of the second time sample of thesignal at the relatively high level of quality using data based on arendition of the second time sample of the signal at the relatively lowlevel of quality.
 20. A method according to claim 18, wherein the secondrendition is derived by performing an upsampling operation on therendition at the relatively low level of quality.